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ABSTRACT 


Systems which are composed of two or more phases, or subsystems, 
arranged in logical sequence are found frequently in industry and 
defense. Standard procedures for computing lower confidence limits 
on reliability of such systems rely on the use of system data. 
Engineering changes to any of these subsystems can effect the invali- 
dation of all existing data, necessitating additional, sometimes 
extensive, testing. Such changes are not infrequent in complex sys- 
tems. A need exists for a method of computing lower confidence limits 
on reliability which uses phase data. Some approximation techniques 
have become available. One such technique is currently being used 
by Applied Physics Laboratory, The Johns Hopkins University. Computer 
simulation techniques are used to analyze the accuracy of this pro- 


cedure. 
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CHAPTER I 


INTRODUCTION 


Current technology has produced a class of systems which are com- 
posed of two or more phases, or subsystems, arranged in logical sequence, 
such that the success of the system depends upon the success of the 
component phases. Thus the reliability of the system, Ro» depends upon 
the reliability of each of the k phases of which it is composed. We 


express this mathematically as follows: 
R, (1) 


where R, is the reliability of the fie phase. Standard statistical 
procedures used to compute lower confidence limits (LCL) on Re 

require the use of system test data. Whenever high degrees of reli- 
ability, at high confidence levels, are demanded of a system, extensive 
testing is required, frequently at considerable cost. 

Engineering changes to any one of the phases may cause the invali- 
dation of all previous system test data. There is a need for a pro- 
cedure to construct lower confidence limits on system reliability using 
phase test data which will salvage as much of the data as possible. 

Approximation procedures are becoming available for so constructing 
these LCL's, one of which is currently being used by Applied Physics 


Laboratory, The Johns Hopkins University. This procedure is discussed 
Pi ieee 

The purpose of this paper is to analyze the accuracy of this pro- 
cedure. Computer simulation techniques will be used to accomplish 


this analysis. 


INumbers in brackets refer to items in the bibliography with the 
same number. 


CHAPTER II 


COMPUTATIONAL PROCEDURE EMPLOYED BY APPLIED PHYSICS LABORATORY 


The APL method for constructing LCL's on R. assumes the following: 


a) mutually independent trials 


aA 


b) the distribution of the reliability estimator, Ros is 


normal. 
The formula for computing the 100(1-a)%, one-sided LCL, Ri is 


R, = R. = P(1-a) 7 oR (2) 


where Ro is the system reliability estimator, P(l-a) is the 


100(1-a) percentile of the standard normal distribution, and oF is 
Ss 

the standard deviation estimator for the R. distribution 

Aw k nw 

Ro = all R, (3) 

i=l 

where Ra is the phase reliability estimator and is defined by 

R. _ s,/n, : (4) 


aw 


’ , .th 
That is, R, denotes the proportion of successes for the i- phase, 


S. > Oue of n, trials. 


The value is computed by 


of 
R, - ——— (5) 


which is an unbiased estimator of the variance of R.- Standard devia- 


tion, ae , is obtained by extracting the square root of the variance. 
S 


Although biased, this estimator is considered to be sufficiently close 


to or el ig 
S 


As stated above, one basic assumption of the model is that R. 


is a normally distributed random variable. As an ancillary result of 


_— 


the simulation, this assumption was verified for those cases for which 
the procedure was reasonably accurate. For each case considered, 1000 
values of R. were computed and plotted on normal probability paper. 
For one such example see Figure 1. For those cases which proved less 


accurate, the distribution departed from normal. 
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CHAPTER III 


THEORY OF THE SIMULATION TECHNIQUE 


One standard interpretation which can be applied to the meaning of 
a lower confidence limit involves the method used to compute it. This 
Says that regardless of the true value of R. the computation will 
produce an LCL, Rs which is less than R. (l-a)% of the time. Use 
can be made of this interpretation in formulating the simulation model. 

If one were to use an exact method of determining R and repeated 
the process several times, he could expect that approximately (l-a)Z 
of the values would be less than Ro: As the number of replications 
approaches infinity, he could expect that exactly (l-a)% of the values 
would be less than Ro 

One could use a similar approach to evaluate the accuracy of an 
unproven method. Given a hypothetical system of known Ras and hence 


aA 


known Ros one could use the proposed method to compute R, on Ro. 
If the method was accurate, approximately (l-a)% of the values so com- 
puted would be less than Ro. If the number of replications was suf- 
ficiently large, the percentile of R. which is less than Re would 
be a good measure of the accuracy of the procedure. This percentile 
will be denoted A,_. 

The computer simulation technique, used herein, follows the above 
approach. The rapid computational ability of the computer allows one 
to simulate the testing process, compute R and replicate the process 
a great many times. One of our measures of effectivess of the method 
will be the relative closeness of Al_y to R.- Other measures of 


effectiveness will be discussed in the results section of this paper. 


The simulation technique begins by reading from data the parameters 
of the case to be considered; i.e., KK, the number of replications de- 
sired (1000 for all cases considered herein); K, the number of phases 
per system; R, and n, for all phases; and oa. Using a pseudo-random 


number generator“ 


and the known phase reliabilities, test data is 
gathered on each of the phases by comparing n, random numbers with 

R. Cha ar K).? Each time the random number generator pro- 
duces a number less than or equal to R. it is counted as a successful 
test. A running total of successes, Sas is kept throughout the course 
of testing. After having so obtained these data, SUBROUTINE RHAT is 
called with parameter a. The values S. and n, are made available to 
RHAT via the DIMENSION statement. SUBROUTINE RHAT returns the values 

R and R.. This operation is repeated for KK replications to compute 
KK values of am and R.. These values are stored in vector matrices 
for later use. 

SUBROUTINE RHAT is a relatively straightforward application of the 
working formulae contained in Chapter 2. One point worthy of note is 
that only ten values of P(l-a) have been read into computer memory in 
order to conserve memory space. The argument (l-a) is digitized to 
facilitate stowage and retrieval. This limits the values of a which 


can be used to increments of .05, from .05 to and including .50, a 


limitation which can easily be changed as the requirements dictate. 


-The pseudo-random number generator used herein is a library sub- 
routine of the Computer Facility, Naval Postgraduate School, Monterey. 
A printout of this subroutine is included in Appendix II. 


3The reader is invited to follow the program logic by referring to 
Appendix II. Extensive use is made of comment cards to explain the 
operations as they are accomplished. The program notation is intended 
to be the same in meaning as that of the body of the paper. It is, of 
necessity, somewhat different in form because of machine limitations. 
As an aid, a glossary of terms is provided in Appendix I. 


Oe 


As the above values of R, are generated, they are summed for 
eventual division by KK to compute their mean. 

Thus far, the simulation has performed the following operations: 

1) generated the statistics S. Ch oe Sere 

2) computed the values for the Ro» i and Al matrices 
(Al being an exact duplication of the R, matrix) 

3) summed the values of R, to get the mean, Me ‘ 

Next, a sorting routine, SUBROUTINE SHSORT,* is called for the 
purpose of putting matrices Ro and Al into ascending numerical 
order. This is done to facilitate plotting these values on probability 
graph paper to determine their distributions. This also facilitates 
obtaining the 100(1-a) percentile of the R, distribution, Ayo" 
Recall the matrix Al was identical to the matrix R, before it was 
ordered. Thus, Ady equals the rac(@les) -2 member of the ordered Al 
matrix. 

Next, the percentile of R, which is less than R. is obtained; 
i.e., the probability Be is less than R_, denoted P(R_). This is 
computed by determining the last element of the ordered Al matrix 
whose value is less than or equal to Re and dividing its index number 


by KK. 


The variance of R oe is calculated next by summing the 
L 


squared differences of R, and a and dividing by KK, the number of 


replications, 
KK 
ye = 
sta ®, 
KK 
“This subroutine is a library subroutine of the Computer Facility, 


Naval Postgraduate School, Monterey, Calif. It is printed out in 
Appendix II. 


<V1= 


The square root is later extracted to obtain the standard deviation, 


i 


The difference between R. and Ay_y is then obtained, which is 


the primary measure of effectiveness of the procedure. 

The remaining statements are editorial in nature; e.g., input/output 
statements which read in data or printout results. Others initialize 
variables, sums, or products to ensure that values are not carried over 
from replication to replication. 

The program output may be summarized as follows: 


a) 1000 values of system reliability estimator, Ro, in 


ascending numerical order 


b) 1000 values of LCL, R > in chronological order 


“a 


c) 1000 values of LCL, Ris in ascending numerical order 


d) system reliability, R, 


e) 100(1-a) percentile of the population of R population, 


Sea 


f) standard deviation of R, population, "R, 


g) probability that R is less than R., P(R,) 


h) mean of R wa 
zr 


i) the difference between R. and A, _y 
i) in.q,, a measure of the amount of testing done and the 


unreliability (q.)> or failure rate 


ee 


CHAPTER IV 


SIMULATION RESULTS 


The accuracy of the procedure is measured by the closeness of Al _y 


relative to R.- A second measure of effectiveness, P(R.); the 
probability that R, is less than Ro» provides a measure of the true 
confidence level achieved by using this procedure. If the method were 
exact, one could expect Al _y to equal R. and (aay to equal (l-a). 
In addition to the above criteria, the mean of R ig and the 
standard deviation, Sia 


ness. These two characteristics provide a measure of the likelihood of 


are also provided as measures of effective- 


the procedure producing a lower confidence limit which differs consid- 
erably from Ro: It can be shown that for those cases for which the 
procedure is reasonably accurate the distribution of R is approxi- 
mately normal. (See Figure 2.) Thus if the procedure is accurate, one 
should expect to find the 1-a percentile to be a known function of 
the mean; e.g., the fo percentile should be approximately 1.28 stand- 
ard deviations above the sample mean. In addition, the mean and vari- 
ance can be used to construct prediction regions into which R, is 
likely to fall, assuming again that R is normally distributed. 

Several cases were examined in evaluation of this method. In each 
case one or more of the parameters, K, R.; n,,» were changed. In 
addition, each case was run for a-values equal to .05, .1, and .2, 
successively. The results of these cases are tabulated in Tables I 
throwehy iil. 

The accuracy of reliability testing procedures is most often a 


function of the amount of testing, Nas and the unreliability of the 


subsystems, q.- This quantity is also tabulated in Tables I through 
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Lower Confidence Limit, R 


=, — 


K 
III and is expressed by ) neq. - Thus Tables I through III provide 


i ee 
criteria upon which one can judge the accuracy of the procedure and an 
effective measure of when to expect this accuracy. If one knows the 
reliabilities for which he is striving and the amount of testing which 
is economically feasible, he has available a good estimate of the 
accuracy of this procedure. 

The cases considered are tabulated in three separate tables ac- 
cording to the purpose for which they were considered. Cases 1 through 
24, Table I, are the same as those considered by Woods and Borsting [2] 
and are intended to provide a direct comparison with their method. 
Table III contains a "family" of cases in which only the parameter n 


a 
changes. These cases demonstrate how the accuracy criteria, Ay _y and 
P(R_); behave as a function of the amount of testing. Figures 3 

through 8 graphically illustrate the behavior of the accuracy criteria 


as a function of ins. for the cases considered in Tables I through 


III. Each of these graphs utilizes one of the three o-levels considered. 
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CHAPTER V 


CONCLUSIONS 


Based upon the results of the simulation, which are tabulated in 


Tables I through III and illustrated in Figures 3 through 8, the fol- 


lowing general conclusions can be drawn: 


a) 


b) 


c) 


The accuracy of the method is a function of mm.q. and, in 
general, will increase as this value increases. However, the 
graphs of the accuracy criteria take the shape of a sawtooth 
curve. Therefore, there is no guarantee that increasing ind 
will improve the accuracy of the method. It is generally true, 
however, that as ind, gets large the difference between 
system reliability, Ro» and Ay _y should converge to zero, 
as should the difference between the desired confidence level, 
(l-a), and P(R.)- 

At confidence levels of .9 and .8, and for values of ind, 
which are sufficiently high, the procedure appears to be rea- 
sonably accurate. Obviously, the numerical value of "suffi- 
ciently high" is subjective and depends upon such unknown param- 
eters as user risk, utility, etc. For most purposes, a value 
of in. q. ~ 4.0 should be sufficiently high at both of the 
above confidence levels. The procedure appears to be inaccu- 
rate if Ss is very high or ind, is small enough to make 

a failure only moderately probable. 

Serious doubt exists as to the acceptability of the method for 
a confidence level of .95 or greater. At the former two con- 


fidence levels there was an apparent convergence toward zero 


error. No such convergence was apparent at the .95 level. 


Boye 


Even in Cases 6 and 8, where n.d. was 22.0 and 23.9, 
respectively, the difference, R. - Al _y: is an order of 


magnitude greater than the corresponding values for a= .1 


and a= .2. Similarly, the actual confidence level is about 


half of that intended. 


a 
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APPENDIX I 


GLOSSARY 


Meaning 


System reliability 
Phase reliability 
System reliability estimator 
Phase reliability estimator 

th 
Successful tests of the i phase 

eh 

Number of tests of the i phase 


Estimator for the variance of the 
reliability estimators 


Estimator for the standard deviation 
of reliability estimators 


Computed lower confidence limit 


The Gea. percentile of the 
computed R, 


Probability that R is less than R. 


The a percentile of the standard 
normal distribution 


Mean of the LCL's generated 
Variance of the LCL's generated 


Standard deviation of the LCL's 
generated 


Number of replications per case 


Number of phases per case 


=a0e 


Notation 


> 


P(R.) 


PKi) 


Q G 


Cait: I 


RR 


Computer Simulation 
Variable Name 


RSUBS 
RSUBI(I) 
RHATS 
RHATI 
Sl) 

AN (T) 


SIGPSQ 


SIG 


RHATL 


AlA 


RSPCN 


P(I) 


MEAN 
VAR 


SIGMA 
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